TransMed: Transformers Advance Multi-Modal Medical Image Classification

Over the past decade, convolutional neural networks (CNN) have shown very competitive performance in medical image analysis tasks, such as disease classification, tumor segmentation, and lesion detection. CNN has great advantages extracting local features of images. However, due to locality convolution operation, it cannot deal with long-range relationships well. Recently, transformers been applied computer vision achieved remarkable success large-scale datasets. Compared natural images, multi-modal images explicit important dependencies, effective fusion strategies can greatly improve deep models. This prompts us study transformer-based structures apply them Existing network architectures require datasets achieve better performance. imaging are relatively small, which makes difficult pure analysis. Therefore, we propose TransMed for classification. combines transformer efficiently extract low-level establish dependencies between modalities. We evaluated our model on two datasets, parotid gland tumors classification knee injury Combining contributions, an improvement 10.1% 1.9% average accuracy, respectively, outperforming other state-of-the-art CNN-based The results proposed method promising tremendous potential be a large number tasks. To best knowledge, this is first work